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Background / Context: 

A common design in education research for interventions operating at a group or cluster 
level is a cluster randomized trial (CRT) (Bloom, 2005). In CRTs, intact clusters (e.g., schools) 
are assigned to treatment conditions rather than individuals (e.g., students) and are frequently an 
effective way to study interventions because they permit researchers to accommodate existing 
school structures and the multilevel nature of many educational interventions (Spybrook & 
Raudenbush, 2009). 

A principal consideration in the design of CRTs is the power with which a study can 
detect effects if they exist (Hedges & Hedberg, 2007; Raudenbush, 1997). To assist in the design 
of CRTs, literature has developed statistical power formulas for a range of cluster based designs 
(e.g., Schochet, 2011; Hedges, Hedberg, & Kuyper, 2012). Based on these frameworks, literature 
has also compiled empirical estimates of the design parameters (e.g., intraclass correlations) to 
describe reasonable values for the design of studies (e.g., Jacobs, Zhu & Bloom, 2010). 

Although CRTs provide rigorous evidence concerning the effect of the treatment on the 
outcome, there is a growing recognition of the need to also investigate the mechanisms through 
which the treatment is presumed to work (e.g., Hong, 2012). One common approach is to 
establish a sequence of structural relationships through a mediation analysis. Mediation analyses 
examine the extent to which a treatment has an indirect effect on an outcome by examining 
changes in a mediator produced by exposure to a treatment and, in turn, how these changes in the 
mediator are associated with changes in outcomes. Despite the importance of mediation analyses 
and the value of well-planned studies, literature is incomplete in describing both power formulas 
for cross-level mediation effects in CRTs and reasonable values for the formula parameters. 
Purpose / Objective / Research Question / Focus of Study: 

In this study, we developed closed form analytic expressions for the test statistics of the 
mediation effect in three level CRT designs examining cross-level (e.g., 3-2-1) mediation. Using 
this test statistic, we estimate the power to detect cross-level mediation effects under different 
scenarios to assist researchers in the planning and design of multilevel mediation studies. 

To fix ideas, consider an example where the intervention is a professional development 
program to increase pedagogical knowledge and it is assigned at random to intact schools (e.g.. 
Caret et al, 2008; 2011). In theory, exposure to the professional development program 
(treatment) should increase teachers’ pedagogical content knowledge (mediator) which in turns 
increases students’ achievement levels (outcome). We might describe the study as a 3-2-1 cross- 
level mediation design because its hypothesized that a treatment assigned at the school level 
(level 3) impacts a classroom level (level 2) mediator which in turn impacts a student level 
(level 1) outcome. The focus of this study was to develop a framework to assess the power to 
detect cross-level mediation for this particular three level CRT design. 

Significance / Novelty of study: 

In detecting cross-level (3-2-1) mediation effects in three level CRTs, literature has 
largely relied on simulation or only the power to detect effects along single paths (e.g., a ox b 
paths) (e.g., Schochet, 2011). By developing closed form expressions of the test statistic for the 
product of cross-level effects, we provide tools for researchers to carefully plan (e.g., the power 
of) their studies so that they are well-positioned to detect mediation effects if they exist. 
Statistical, Measurement, or Econometric Model: 

We investigated two situations in our study. In the first situation, we examined 3-2-1 
cross-level mediation effects as they pertain to the second (teacher) level only. In contrast, in the 
second situation we examined estimation of both level two (teacher) and level three (school) 
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mediation effects. We assumed a fully balanced design (i.e., equal number of students/classroom, 
classrooms/school, and equal allocation of school treatment and control conditions) and did not 
consider variables that might confound the mediator-outcome relationship. Although the 
treatment-mediator path is unconfounded because of random assignment, the mediator-outcome 
path may be confounded because levels of the mediator were not randomly assigned. We briefly 
discuss this and other limitations and extensions in the Conclusion section. 

Situation 1 . Mediation effects have been conventionally formulated using a linear 
structural equation framework (e.g., Baron & Kenny, 1986) (Figure 1). For instance, consider the 
following set of linear equations 






■^0 jk ~ PoQk ■ 
Pmk ~ Ytm~ 


ybM.,- 
■ YmJ'k 


“Ojt 


( 2 ) 


In expression (1), the first equation describes the relationships of teachers within schools where 
Mjk is a continuously distributed mediator for teacher j in school k, is the intercept for the 


mediator equation and is the normally distributed error term with variance (j^^„ . The second 


equation describes school-level relationships where y" is the grand intercept, is the treatment 
with associated coefficient a describing its relationship with the mediator, and is a normally 
distributed error term with variance . In expression (2), Y... is the outcome for student i 

served by teacher j in school k, is the intercept, sl^. is the normally distributed error term with 
variance is the teacher intercept, is the mediator with associated coefficient b 


capturing the conditional association between the mediator and the outcome (which has 
coefficient y), ulj^ is a normally distributed teacher random effect term with variance is 


the school level intercept, is the treatment indicator with associated coefficient , and 


is a normally distributed school random effect term with variance . 

'b 


One approach to estimating the indirect effect using this system of equations involves 
fitting a sequence of multilevel models to expressions (1) and (2) (e.g., Zhang, Zyphur, & 
Preacher, 2009) so that the mediation effect is the product of the direct effect of the treatment of 
interest on the mediator (a in expression (1)) by the direct effect of the mediator on the outcome 
controlling for the treatment (b in expression (2)). 

The asymptotic variance of the (single) mediation effect can be estimated using the Delta 
method (Sobel, 1982; Bollen, 1987; 1989), 
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In turn, the significance of the mediation effect in large samples can be approximated using 

ab 
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where z„has an approximate normal distribution in large samples (Sobel, 1982). In the context of 


a three-level CRT with treatment assigned to schools and a level two mediator, the variances of 
the a and b estimated paths are 
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where ni, H 2 , and nj are the student, teaeher, and sehool sample sizes. We ean write the test 
statistie for the mediation effeet as 


ab 


2m =■ 


(6) 
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As a result, similar to power analyses for deteeting main effeets in CRTs, given a priori estimates 
of the standardized varianee eomponents and the expeeted magnitude of the path eoefficients, we 
ean estimate the power of a given design. 

Situation 2. In the seeond situation we examined seenarios in whieh the within-group 
effects differ from the between-group effects (Zhang, Zyphur, & Preacher, 2009). We adapt 
equations (1) and (2) to now include cluster (school) level mediator means such that 
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Here we continue with the aforementioned notation and introduce by^ as the within-group effect, 
bg as the between-group effect, and M.j, as mean of the mediator for school k such that mediator 
values for teachers are centered on their school means (i.e., group-mean centered). Because 
group-mean centered variables at the teacher level are independent of the school level means, the 
power for the within-group (teacher) mediation effect ( A ) can be estimated using the steps 
outlined above. 

To estimate power for the between-group (school) mediation effect ( b^ ), we begin by 
noting that an unbiased estimate of a in equation 8 can had using an OLS estimator on the 
aggregated data so that 




An unbiased estimate of a and its standard error can be obtained using 
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where and Oj are the standard deviations of the aggregated mediator and treatment 


variables, is their correlation. Similarly, we can estimate b^ using (note that although M 
and M.j, are independent, r/ and M.j, are not independent) 

= Zooo + Zoo a/ + (11) 

Where and its standard error are 
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where is the partial correlation such that 




W,M. Py,T,Pm,T^ 


(13) 


SREE Spring 2014 Conference Abstract Template 


6 



Next we note based on above the statistieal significanee of the between mediation effect, abg , 
denoted as , can be estimated using 


ab„ 
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Similarly, we can write the t-values of the a and Bb parameters in (10) and (12) as 
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With some algebra we find that (assuming positive mediation) 
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Using expressions (10) and (12) we can rewrite this as 
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This expression can be used to estimate the power with which we can detect a between group 
mediation effect if it exists. We could also parameterize the expressions in terms of a and bg 
using, for example the following relationships 
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and 

Usefulness / Applicability of Method: 

Using data from a large study of many different professional development programs, we applied 
the method to design studies examining the extent to which teachers’ knowledge mediates the 
relationship between participation in a professional development program and students’ 
cognitive development. Our application examines designs that achieve 80% power based on 
design parameters (e.g., ICC, effect size) from this study as well as other studies. 


Findings / Results: 

Our results (not presented due to space constraints) suggested that studies investigating cross- 
level mediation using CRTs will typically need large sample sizes or large mediation effects to 
have an 80% chance of detecting mediation effects if they exist. 


Conclusions: 

The method developed allows researchers to plan and design more effective studies of cross- 
level mediation. Our analysis of 3-2-1 mediation involving professional development, teacher 
knowledge, and student achievement underscored the importance of carefully planning and 
designing studies to detect effects. Studies with typical CRT sample sizes (e.g., 40 schools) will 
generally not be sufficient to detect mediation in this context. The framework we developed has 
several limitations. For instance, our framework assumes linearity, large sample approximations, 
sequential ignorability, that mediation is not moderated, mediator slopes do not randomly vary 
across schools, and it does not include latent variables. Subsequent works addresses some of 
these shortcomings. 
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